TRG 11 - Testing and Quality (TRG Suggestion) #1396
Conversation
Moved code coverage TRG position to a new section TRG-11. Adapted and reworded suggestions and removed threshold requirement
Fixed anchor linking
ds-hzimmer
changed the title
ds-hzimmer
requested review from
a team,
MonikaJacobsen,
lgblaumeiser,
matbmoser,
stephanbcbauer and
ther3sa
lgblaumeiser
left a comment
lgblaumeiser
left a comment
There was a problem hiding this comment.
I have to admit, I do not get the reason for this TRG. It is missing:
- A reason on why this TRG is needed
- Concrete guidelines on what is expected
- Requires an external visible impact, the other TRGs are all concerned with general expectations an end user can expect so that he can adopt the project easier. I do not see, how this trg adds such a quality, as the adoptors user experience is not influenced by the code coverage of our products, but by the quality experience of the software. Code coverage is an internal view for a developer group, if this helps them to improve the quality experience.
In this form, the TRG imho does not make sense, sorry, to be so direct.
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| | Status | Created | Post-History | |
There was a problem hiding this comment.
Change logs are always latest change on top!
|
|
||
| ## Why | ||
|
|
||
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. |
There was a problem hiding this comment.
That is not a goal, it is a statement that you want that, but what do we want to achieve with this? Actually, I think, that is quite a good question, right? I do not see an overall why that can be easily expressed, to be honest.
|
|
||
| 1.1. **Code Coverage** | ||
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. |
There was a problem hiding this comment.
Please use SHOULD or MUST or MAY for expressing things that have to be done!
And is it should or must? And what does report mean?
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. | ||
| - Code coverage measurement and reporting should be automated and integrated into the applicable CI/CD pipelines as part of the release process checks. | ||
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | ||
| - However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value. |
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. | ||
| - Code coverage measurement and reporting should be automated and integrated into the applicable CI/CD pipelines as part of the release process checks. | ||
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). |
There was a problem hiding this comment.
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | |
| - An Eclipse Tractus-X software project **MAY** use a line coverage of 80% as is the recommendation of the Tractus-X Test and Release Management, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). |
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | ||
| - However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value. | ||
| - Thus project leads respectively the committers of a project define, document and enforce acceptable coverage levels and any project-specific Quality Gates themselves. | ||
| - Per default, code coverage should be calculated using both unit tests and integration tests combined, unless a project follows a different documented approach. |
There was a problem hiding this comment.
Again this is a SHOULD
| - Is code only used for logging, metrics, or monitoring. | ||
| - Depend on platform-specifics, for example hardware that cannot be simulated in a test. | ||
|
|
||
| Extensive or critical deliberate exclusions from the code coverage measurement should be reviewed, documented and approved by the project's committers. |
There was a problem hiding this comment.
This is redundant, they have to be documented, this implies a review by the committers
|
|
||
| ### 3. Quality Assurance | ||
|
|
||
| 3.1. **Code Review Requirements** |
There was a problem hiding this comment.
This section is basically redundant.
|
|
||
| ### 2. Analysis and Reporting | ||
|
|
||
| 2.1. **Tools for Code Coverage Reporting** |
There was a problem hiding this comment.
This section does not add much value, either you express a requirement or you give a concrete guideline, but this is neither.
| - During review, the project's committers may also accept changes that reduce code coverage below this defined threshold value if they explicitly decide this. | ||
| - As a general guidance, as outlined above new code is recommended to reach at least 80.0% coverage (SonarQube default Quality Gate). This is not mandatory. | ||
|
|
||
| 3.2. **Risk Analysis** |
There was a problem hiding this comment.
That does not add value. In general, the committers of a Tractus-X project are responsible for their quality. They decide on the real existing quality whether to release, not because of a metric.
stephanbcbauer
left a comment
stephanbcbauer
left a comment
There was a problem hiding this comment.
Nothing to add additional to @lgblaumeiser comments, just my stupid question about projects/products
|
|
||
| 1.1. **Code Coverage** | ||
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. |
There was a problem hiding this comment.
Never thought about it, but are we talking about projects within Eclipse Tractus-X? Or products?
There was a problem hiding this comment.
That is a good point, I also stumbled over this, as far as I remember. In general Eclipse Tractus-X is an Open Source project, and contains lots of repos. We added this level of product to identify lead repositories which might have dependent repos.
3 participants
Suggestion for a Tractus-X Release Guideline (TRG) for "Testing and Quality" in a new section TRG-11, with an initial TRG 11.01 - Code Coverage
Revised from previous pull request (closed) due to some past reviewers no longer being active in the project, adaptations to incorporate their objections (reworded definition, removed threshold requirement), and move needed to a new section TRG-11 is now already occupied by the new KITs section.
Description
Tractus-X Release Guidelines (TRGs): A new TRG section for "Testing and Quality" should be created, and this TRG is a first suggestion in this new section.
The goal is to ensure that all released Eclipse Tractus-X software components meet sufficient test coverage to guarantee quality, stability, and reliability. Measurement and reporting should be performed by all Tractus-X projects. A minimum quality threshold is suggested by Test and Release Management based on default Eclipse Tractus-X settings, but not required or automatically enforced.
The coverage should be measured and monitored using established Open Source tools. For reporting of the measured values, Eclipse Tractus-X already provides a SonarQube Cloud installation. https://sonarcloud.io/organizations/eclipse-tractusx/projects
Other tools to measure code coverage and report it in the build pipeline are also possible if decided by a project. Using either alternative tools or other SonarQube installations should also be the method if the code of a Tractus-X repository is branched to a new repository outside of the Eclipse Tractus-X project, and thus a team would not have access to this SonarCloud organization.
Pull request created by issue:
eclipse-tractusx/sig-release#970
Pre-review checks
Please ensure to do as many of the following checks as possible, before asking for committer review:
[x] DEPENDENCIES are up-to-date. Dash license tool. Committers can open IP issues for restricted libs.
[x] Copyright and license header are present on all affected files